feat(prose-tests): record everything the agents do, results included - #557
Merged
Conversation
This was referenced Jul 26, 2026
Merged
leeovery
force-pushed
the
prose-tests/consequence-claims
branch
from
July 27, 2026 14:32
4790742 to
959a33a
Compare
leeovery
force-pushed
the
prose-tests/record-everything
branch
from
July 27, 2026 14:32
d4c5490 to
f09d886
Compare
leeovery
force-pushed
the
prose-tests/consequence-claims
branch
from
July 27, 2026 14:34
959a33a to
5701d7f
Compare
leeovery
force-pushed
the
prose-tests/record-everything
branch
from
July 27, 2026 14:34
f09d886 to
499e06a
Compare
leeovery
force-pushed
the
prose-tests/consequence-claims
branch
from
July 27, 2026 14:37
5701d7f to
97069a9
Compare
leeovery
force-pushed
the
prose-tests/record-everything
branch
from
July 27, 2026 14:37
499e06a to
5ff1980
Compare
leeovery
force-pushed
the
prose-tests/consequence-claims
branch
from
July 27, 2026 14:39
97069a9 to
7047929
Compare
leeovery
force-pushed
the
prose-tests/record-everything
branch
from
July 27, 2026 14:39
5ff1980 to
227b88d
Compare
leeovery
force-pushed
the
prose-tests/consequence-claims
branch
from
July 27, 2026 14:41
7047929 to
75d4b1c
Compare
leeovery
force-pushed
the
prose-tests/record-everything
branch
from
July 27, 2026 14:41
227b88d to
cdc4a72
Compare
leeovery
force-pushed
the
prose-tests/consequence-claims
branch
from
July 27, 2026 14:43
75d4b1c to
e50a157
Compare
leeovery
force-pushed
the
prose-tests/record-everything
branch
from
July 27, 2026 14:43
cdc4a72 to
dab60ef
Compare
leeovery
force-pushed
the
prose-tests/consequence-claims
branch
from
July 27, 2026 14:45
e50a157 to
1f8a100
Compare
leeovery
force-pushed
the
prose-tests/record-everything
branch
from
July 27, 2026 14:45
dab60ef to
94f3009
Compare
leeovery
force-pushed
the
prose-tests/consequence-claims
branch
from
July 27, 2026 14:47
1f8a100 to
613128f
Compare
leeovery
force-pushed
the
prose-tests/record-everything
branch
from
July 27, 2026 14:47
94f3009 to
510908b
Compare
leeovery
force-pushed
the
prose-tests/consequence-claims
branch
from
July 27, 2026 14:49
613128f to
0ebdd03
Compare
leeovery
force-pushed
the
prose-tests/record-everything
branch
from
July 27, 2026 14:49
510908b to
16f1664
Compare
leeovery
force-pushed
the
prose-tests/consequence-claims
branch
from
July 27, 2026 14:51
0ebdd03 to
5cf48e8
Compare
leeovery
force-pushed
the
prose-tests/record-everything
branch
from
July 27, 2026 14:51
16f1664 to
7222ed8
Compare
leeovery
force-pushed
the
prose-tests/consequence-claims
branch
from
July 27, 2026 15:42
5cf48e8 to
2c1fe91
Compare
leeovery
force-pushed
the
prose-tests/record-everything
branch
from
July 27, 2026 15:42
7222ed8 to
ffd9dfc
Compare
The action log captured a command's input and never its result, and a walker exploited exactly that gap: it claimed "the snapshot's MENU section is empty", the asserter had no way to check, and four consistent runs agreed on something untrue. Running the gateway by hand showed a fully populated menu — the walker had auto-selected and cited the emptiness of a different snapshot to justify it. So the hook now records the lot. Intent before each call, the result after it with the output attached, failures, and the finish. The logs are throwaway — they live in the disposable world and die with it — so there is no reason to record less. All three agents are instrumented, each for what it can get wrong: - the walker, on every tool event, into its world - the orchestrator, on Bash, so its own steps are visible - the asserter, on any tool call at all — it is contracted to use none, so a call is a breach and lands in a repo-local log rather than vanishing The asserter is told the record now settles claims about output: where its account of what a command returned disagrees with what the record says came back, the record wins and the discrepancy is itself a finding. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
leeovery
force-pushed
the
prose-tests/record-everything
branch
from
July 27, 2026 15:44
ffd9dfc to
d5b6cfe
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
gateway.cjs view, not the bare selection call).PreToolUse), the result after it with the output attached (PostToolUse), failures (PostToolUseFailure), and the finish (Stop). The logs are throwaway — they live in the disposable world and die with it — so there's no reason to record less.prose-walkerprose-orchestratorprose-asserterTest plan
npm testgreen: 1699 tests, 0 fail.continue-*cases that failed on the false claim should now be judged against what the gateway actually returned — worth re-running.🤖 Generated with Claude Code
Stack